feat: Connect DYN_LOG level to TLLM_LOG_LEVEL #3451

tanmayv25 · 2025-10-07T00:36:27Z

Overview:

This fixes the logging within TRTLLM engine. The code will set TLLM_LOG_LEVEL=INFO by default which would allow the print_iter_log to appear.

Without this TRTLLM logs will not appear during inference.

Before


data: {"id":"chatcmpl-d043cc61-c996-4baf-b2cb-7b9b323f59b9","choices":[{"index":0,"delta":{"content":" me","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759789572,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

data: {"id":"chatcmpl-d043cc61-c996-4baf-b2cb-7b9b323f59b9","choices":[{"index":0,"delta":{"content":" start","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759789572,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

data: {"id":"chatcmpl-d043cc61-c996-4baf-b2cb-7b9b323f59b9","choices":[{"index":0,"delta":{"content":" by","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759789572,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

data: {"id":"chatcmpl-d043cc61-c996-4baf-b2cb-7b9b323f59b9","choices":[{"index":0,"delta":{"content":" recalling","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null},"finish_reason":"length"}],"created":1759789572,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

data: [DONE]

After

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 94, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.382040023803711ms, prev_device_step_time = 2.4019839763641357ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":".","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 95, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.3653507232666016ms, prev_device_step_time = 2.436768054962158ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":" Let","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 96, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.4323463439941406ms, prev_device_step_time = 2.398848056793213ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":" me","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 97, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.274036407470703ms, prev_device_step_time = 2.4051520824432373ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":" start","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 98, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.378702163696289ms, prev_device_step_time = 2.430624008178711ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":" by","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null}}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

[10/07/2025-00:29:01] [TRT-LLM] [I] iter = 99, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 2.4247169494628906ms, prev_device_step_time = 2.4163520336151123ms, timestamp = 2025-10-07 00:29:01, num_scheduled_requests: 1, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 1}
data: {"id":"chatcmpl-51e76f86-db3a-4092-bf60-94372ccaf7b4","choices":[{"index":0,"delta":{"content":" recalling","function_call":null,"tool_calls":null,"role":"assistant","refusal":null,"reasoning_content":null},"finish_reason":"length"}],"created":1759796941,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":null}

data: [DONE]

Summary by CodeRabbit

New Features
- Automatically aligns TensorRT-LLM logging level with your existing DYN_LOG setting at startup for consistent logs.
- Respects any pre-set TLLM_LOG_LEVEL and falls back to sensible defaults when unspecified.
- Supports comma-delimited DYN_LOG values and maps them to appropriate TensorRT-LLM levels.
- Integrates with the runtime logging setup by default; can be disabled via an environment flag.

coderabbitai · 2025-10-07T00:40:17Z

Walkthrough

Introduces pre-import configuration to set TLLM_LOG_LEVEL from DYN_LOG before importing TensorRT-LLM. Adds mapping and configuration functions to the logging module, integrates TensorRT-LLM logging setup into existing configure_dynamo_logging, and respects overrides via existing environment variables or a skip flag.

Changes

Cohort / File(s)	Summary
TRT-LLM pre-import env setup `components/src/dynamo/trtllm/main.py`	Adds early environment configuration to set `TLLM_LOG_LEVEL` from `DYN_LOG` (via safe mapping) before any `tensorrt_llm` imports, ensuring logging level is established prior to initialization.
Logging mapping and integration `lib/bindings/python/src/dynamo/runtime/logging.py`	Adds `map_dyn_log_to_tllm_level(...)` to translate `DYN_LOG` to TensorRT-LLM levels; adds `configure_trtllm_logging(...)`; updates `configure_dynamo_logging` to invoke TRT-LLM logging unless `DYN_SKIP_TRTLLM_LOG_FORMATTING` is set; respects preexisting `TLLM_LOG_LEVEL`.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Env as Process Env
  participant Main as dynamo/trtllm/main.py
  participant L as logging.py
  participant TRT as tensorrt_llm

  Note over Main: Module import start
  Main->>Env: Check TLLM_LOG_LEVEL
  alt Not set
    Main->>L: map_dyn_log_to_tllm_level(DYN_LOG)
    L-->>Main: TLLM level string
    Main->>Env: Set TLLM_LOG_LEVEL
  else Already set
    Note over Main: Preserve existing value
  end
  Main->>TRT: import tensorrt_llm...
  Note over TRT: Initializes with established log level

sequenceDiagram
  autonumber
  participant App as Application
  participant Log as configure_dynamo_logging
  participant L as logging.py
  participant Env as Process Env

  App->>Log: configure_dynamo_logging(...)
  alt DYN_SKIP_TRTLLM_LOG_FORMATTING not set
    Log->>L: configure_trtllm_logging(dyn_level)
    L->>Env: If TLLM_LOG_LEVEL unset, set from map_dyn_log_to_tllm_level(DYN_LOG)
  else Skip flag set
    Note over Log,Env: Do not modify TLLM_LOG_LEVEL
  end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A hare taps logs before the dawn,
Sets TLLM’s voice, then hops along.
From DYN’s whisper, levels bloom—
INFO, WARN, or VERBOSE plume.
Pre-import seeds the forest bright,
So Tensor trees log true and light. 🐇✨

Pre-merge checks

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description Check	⚠️ Warning	The pull request description only includes the Overview section and provides before-and-after examples, but it omits the required Details, Where should the reviewer start?, and Related Issues sections defined in the repository template, so it does not conform to the template.	Please update the description to include a “#### Details:” section summarizing the changes made, a “#### Where should the reviewer start?” section pointing to the key files, and a “#### Related Issues:” section linking any relevant GitHub issues.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title succinctly describes the primary change by indicating that the DYN_LOG level is now being connected to the TLLM_LOG_LEVEL environment variable, matching the core functionality introduced in the diff. It uses clear and specific terminology without extraneous details or noise.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d809906 and 35a5303.

📒 Files selected for processing (2)

components/src/dynamo/trtllm/main.py (1 hunks)
lib/bindings/python/src/dynamo/runtime/logging.py (2 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

components/src/dynamo/trtllm/main.py (1)

lib/bindings/python/src/dynamo/runtime/logging.py (1)

map_dyn_log_to_tllm_level (198-222)

🪛 Ruff (0.13.3)

lib/bindings/python/src/dynamo/runtime/logging.py

225-225: Unused function argument: dyn_level

(ARG001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

GitHub Check: trtllm (amd64)
GitHub Check: Build and Test - dynamo

components/src/dynamo/trtllm/main.py

lib/bindings/python/src/dynamo/runtime/logging.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Tanmay Verma <[email protected]>

lib/bindings/python/src/dynamo/runtime/logging.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Tanmay Verma <[email protected]>

rmccorm4

Thanks Tanmay! Just needs pre-commit fixed.

Signed-off-by: Tanmay Verma <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Piotr Tarasiewicz <[email protected]>

tanmayv25 added 2 commits October 6, 2025 17:33

feat: Connect DYN_LOG level to TLLM_LOG_LEVEL

00810a7

Fix format0

35a5303

tanmayv25 requested review from a team as code owners October 7, 2025 00:36

pull-request-size bot added the size/M label Oct 7, 2025

github-actions bot added the feat label Oct 7, 2025

coderabbitai bot reviewed Oct 7, 2025

View reviewed changes

components/src/dynamo/trtllm/main.py Outdated Show resolved Hide resolved

lib/bindings/python/src/dynamo/runtime/logging.py Show resolved Hide resolved

Apply suggestion from @coderabbitai[bot]

1d12897

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Tanmay Verma <[email protected]>

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 18:53 Inactive

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 18:59 Inactive

rmccorm4 reviewed Oct 7, 2025

View reviewed changes

lib/bindings/python/src/dynamo/runtime/logging.py Outdated Show resolved Hide resolved

Update lib/bindings/python/src/dynamo/runtime/logging.py

9b26640

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Tanmay Verma <[email protected]>

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 19:31 Inactive

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 19:35 Inactive

Fix the log levels

ab4bf2a

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 20:01 Inactive

tanmayv25 requested a review from rmccorm4 October 7, 2025 20:10

rmccorm4 approved these changes Oct 7, 2025

View reviewed changes

Fix log

372f92c

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 20:12 Inactive

copy-pr-bot bot temporarily deployed to GITLAB October 7, 2025 20:13 Inactive

tanmayv25 enabled auto-merge (squash) October 7, 2025 20:13

tanmayv25 merged commit 68bbe8a into main Oct 7, 2025
19 of 20 checks passed

tanmayv25 deleted the tanmayv-log branch October 7, 2025 21:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Connect DYN_LOG level to TLLM_LOG_LEVEL #3451

feat: Connect DYN_LOG level to TLLM_LOG_LEVEL #3451

Uh oh!

tanmayv25 commented Oct 7, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rmccorm4 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Connect DYN_LOG level to TLLM_LOG_LEVEL #3451

feat: Connect DYN_LOG level to TLLM_LOG_LEVEL #3451

Uh oh!

Conversation

tanmayv25 commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview:

Before

After

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

Pre-merge checks

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rmccorm4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tanmayv25 commented Oct 7, 2025 •

edited

Loading

coderabbitai bot commented Oct 7, 2025 •

edited

Loading